feat: Add third-party evaluator CLI integration (DeepEval + Autoevals) by stone-coding · Pull Request #1779 · aws/agentcore-cli

stone-coding · 2026-07-16T23:17:28Z

Summary

Add --param key=value (repeatable) and --parameters-file for metric constructor kwargs
Registry-driven architecture: adding a new library = one config entry + one template directory
Library-specific defaults: DeepEval (300s/1024MB), Autoevals (60s/512MB)
Per-metric warnings for metrics requiring retrieval_context or expected_output
Templates generate: lambda_function.py + pyproject.toml + execution-role-policy.json
E2E lifecycle test: create → add 3P evaluator → deploy → invoke → run eval

Test plan

62 unit tests passing (npx vitest run src/cli/primitives/__tests__/EvaluatorPrimitive.test.ts)
E2E test: e2e-tests/third-party-eval-lifecycle.test.ts (DeepEval + Autoevals full lifecycle)
Manual E2E: 4/4 evaluators deployed and scored successfully against live AgentCore service

Customer experience

# Before: ~40 lines of handler code + SAM/CDK template + manual deployment
# After: 2 commands
agentcore add evaluator --3p-library deepeval --metric FaithfulnessMetric --param threshold=0.7 --name faithfulness --level TRACE
agentcore deploy --yes

…t tests

…Eval + Autoevals)

…support - DeepEval template: add /tmp workaround + DEEPEVAL_TELEMETRY_OPT_OUT for Lambda - Autoevals template: fix scorer= → metric= to match SDK API - Rename --from-3p-library → --3p-library - Add --param key=value (repeatable) for metric constructor kwargs - Add --parameters-file for JSON file of kwargs - Add parseParamFlags() utility + unit tests - --param and --parameters-file are mutually exclusive

- Add e2e-tests/third-party-eval-lifecycle.test.ts: full lifecycle test for DeepEval + Autoevals 3P evaluators (create → add → deploy → invoke → run eval) - Fix autoevals template: params go to AutoevalsAdapter (not metric constructor) - Add openai>=1.0.0 to autoevals pyproject.toml (required by LLM-based scorers)

…K rename)

chianhw · 2026-07-22T08:00:27Z

+)
+from bedrock_agentcore.evaluation.custom_code_based_evaluators.third_party.autoevals import AutoevalsAdapter
+
+adapter = AutoevalsAdapter(metric={{ EvaluatorClass }}(), {{{ EvaluatorParams }}})


AutoEvalsAdapter?

chianhw · 2026-07-22T08:01:25Z

                if (cliOptions.instructions) fail('--instructions cannot be used with --type code-based');
                if (cliOptions.ratingScale) fail('--rating-scale cannot be used with --type code-based');
              }



We don't handle evalType === 'llm-as-a-judge' && threePLibrary?

Add validation guard so that explicitly passing --type llm-as-a-judge with --3p-library fails with a clear error message instead of silently proceeding into an invalid state.

Copilot

Pull request overview

This PR extends AgentCore CLI evaluator support by adding a registry-driven integration for third-party evaluator libraries (DeepEval and Autoevals), including new CLI flags for metric selection and constructor parameters, and new template assets for scaffolding.

Changes:

Added --3p-library, --metric, repeatable --param, and --parameters-file (plus --memory) for third-party evaluator scaffolding.
Introduced a registry (THIRD_PARTY_EVALUATOR_LIBRARIES) with per-library defaults (timeout/memory) and per-metric runtime warnings.
Added new evaluator template assets (DeepEval/Autoevals) and an E2E lifecycle test covering create → add → deploy → invoke → run eval.

Reviewed changes

Copilot reviewed 13 out of 14 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
src/schema/schemas/primitives/evaluator.ts	Extends managed code-based evaluator schema to optionally accept Lambda memory size.
src/schema/llm-compacted/agentcore.ts	Mirrors the new optional `memorySizeMb` field in the compacted TS interface.
src/cli/templates/EvaluatorRenderer.ts	Adds a renderer for third-party evaluator templates with structured template data.
src/cli/primitives/EvaluatorPrimitive.ts	Implements third-party evaluator registry, CLI parsing/validation, config generation, and template scaffolding.
src/cli/primitives/tests/EvaluatorPrimitive.test.ts	Adds unit tests for registry defaults, config generation, and param parsing helpers.
src/assets/evaluators/deepeval-lambda/*	Adds DeepEval evaluator scaffold (lambda handler + pyproject + IAM policy).
src/assets/evaluators/autoevals-lambda/*	Adds Autoevals evaluator scaffold (lambda handler + pyproject + IAM policy).
src/assets/tests/snapshots/assets.snapshot.test.ts.snap	Updates asset snapshot listing to include new evaluator template directories.
e2e-tests/third-party-eval-lifecycle.test.ts	Adds an E2E test covering DeepEval + Autoevals evaluator lifecycle (skipped when env can’t run).
.gitignore	Ignores `test-e2e-output/` directory.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

+export function jsonToKwargs(json: string): string {
+  const obj = JSON.parse(json) as Record<string, unknown>;
+  return Object.entries(obj)
+    .map(([key, value]) => `${key}=${jsonToPythonValue(value)}`)
+    .join(', ');
+}
+
+export function parseParamFlags(params: string[]): string {
+  return params
+    .map(param => {
+      const eqIndex = param.indexOf('=');
+      if (eqIndex === -1) {
+        throw new Error(`"${param}" is not in key=value format`);
+      }
+      const key = param.slice(0, eqIndex);
+      const rawValue = param.slice(eqIndex + 1);
+      let value: unknown;
+      try {
+        value = JSON.parse(rawValue);
+      } catch {
+        value = rawValue;
+      }
+      return `${key}=${jsonToPythonValue(value)}`;
+    })
+    .join(', ');
+}


+              if (threePLibrary) {
+                if (!cliOptions.metric) fail('--metric is required when using --3p-library');
+                if (cliOptions.model) fail('--model cannot be used with --3p-library');
+                if (cliOptions.instructions) fail('--instructions cannot be used with --3p-library');
+                if (cliOptions.ratingScale) fail('--rating-scale cannot be used with --3p-library');
+                if (cliOptions.lambdaArn) fail('--lambda-arn cannot be used with --3p-library');
+                if (cliOptions.config) fail('--config cannot be used with --3p-library');
+              }


+  private buildThirdPartyConfig(
+    name: string,
+    libraryConfig: ThirdPartyLibraryConfig,
+    timeoutStr?: string,
+    memoryStr?: string
+  ): EvaluatorConfig {
+    const timeoutSeconds = timeoutStr ? parseInt(timeoutStr, 10) : libraryConfig.defaultTimeoutSeconds;
+    const memorySizeMb = memoryStr ? parseInt(memoryStr, 10) : libraryConfig.defaultMemorySizeMb;
+    return {
+      codeBased: {
+        managed: {
+          codeLocation: `app/${name}/`,
+          entrypoint: DEFAULT_CODE_ENTRYPOINT,
+          timeoutSeconds,
+          memorySizeMb,
+          additionalPolicies: ['execution-role-policy.json'],
+        },
+      },
+    };
+  }


haomiao037 added 4 commits July 16, 2026 16:04

Add deepeval CLI integration: flags, templates, warnings, memory, uni…

6fba5c8

…t tests

Refactor CLI to registry-driven third-party evaluator framework (Deep…

d44430a

…Eval + Autoevals)

stone-coding requested a review from a team July 16, 2026 23:17

github-actions Bot added the size/xl PR size: XL label Jul 16, 2026

fix: Rename AutoevalsAdapter → AutoEvalsAdapter in template (match SD…

9153b3c

…K rename)

github-actions Bot added size/xl PR size: XL and removed size/xl PR size: XL labels Jul 20, 2026

chianhw reviewed Jul 22, 2026

View reviewed changes

fix: Reject --3p-library with --type llm-as-a-judge

7b5348c

Add validation guard so that explicitly passing --type llm-as-a-judge with --3p-library fails with a clear error message instead of silently proceeding into an invalid state.

Copilot AI review requested due to automatic review settings July 22, 2026 18:54

github-actions Bot added size/xl PR size: XL and removed size/xl PR size: XL labels Jul 22, 2026

Copilot started reviewing on behalf of stone-coding July 22, 2026 18:54 View session

Copilot AI reviewed Jul 22, 2026

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

feat: Add third-party evaluator CLI integration (DeepEval + Autoevals)#1779

feat: Add third-party evaluator CLI integration (DeepEval + Autoevals)#1779
stone-coding wants to merge 6 commits into
aws:mainfrom
stone-coding:deepeval-cli-integration

stone-coding commented Jul 16, 2026 •

edited

Loading

Uh oh!

chianhw Jul 22, 2026

Uh oh!

chianhw Jul 22, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Uh oh!

Conversation

stone-coding commented Jul 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Test plan

Customer experience

Uh oh!

chianhw Jul 22, 2026

Choose a reason for hiding this comment

Uh oh!

chianhw Jul 22, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

stone-coding commented Jul 16, 2026 •

edited

Loading